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ABSTRACT 

A study investigated whether students who are given 
experience in writing would improve the quality of their writing over 
the course of an academic year, and whether the amount of improvement 
would be associated with the amount of instructional writing 
experience across the curriculum. A 50-minute essay exam designed to 
assess writing skills was administered at the beginning of the fall 
term and again at the end of the spring term to 113 students in 
composition classes at 3 colleges in the Minnesota Community College 
System. Results showed that students given experience in writing 
improved the quality of their writing over the course of the academic 
year (with the gain quite consistent across grade-point-average 
groups), and that gains in writing quality increased according to the 
amount of instrucvional writing experience. A pilot study explored a 
subsidiary hypothesis that the quality of instructional activities 
would also contribute to gains in the quali'./ of students' writing. 
Findings suggest that the quality of writing assignments is likely to 
be more important than the number of writing assignments students are 
given. Further writing across the curriculum research should 
concentrate on identifying features of assignment design which 
optimize student learning, and on developing a systematic procedure 
for rating the quality of instructional assignments based on these 
features. (Three taoles of data are included; the essay question and 
a holistic scoring guide are attached.) (SR) 
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Do students who experience instructional writing across the curriculum improve the 
quality of their writing over the course of an academic year? If so, is this improvement 
related to the amount of instructional writing they receive? Is it related to the quality of 
instructional writing assignments and their integration into other course activities? This 
presentation will describe a longitudinal study conducted in the Minnesota Community 
College System designed to answer these questions. The study was part of an evaluation 
of a Writing Across the Curriculum (WAC) faculty development program. 

A major assumption of WAC is that, by writing in a variety of classes, students will 
become better writers than it their experience with writing activities is limited to classes 
in English and composition. 1 he theory underlying this assumption is that, by writing in 
a number of disciplines, students will do more writing, and will write using different 
formats and with different purposes, than they would otherwise; and that this increase in 
the amount and variety of writing will result in an improvement in the quality of writing. 
Since writing competence may improve slowly, students were tested for the cumulative 
effects of writing over the course of an academic year. 

The study was conducted to test the hypotheses that students who are given experience 
in writing: 1) will improve the quality of their writing over the course of an academic 
year; and 2) the amount of improvement in the quality of their writing will be associated 
with their amount of instructional writing experience across the curriculum. In addition, 
a subsidiary pilot study was conducted to explore the hypothesis that the quality of 
instructional writing activities also correlates with improvement in the quality of writing. 
The pilot was conducted because cross-sectional and case study results obtained earlier in 
the grant suggested that it is not the amount of writing, but the design and/or 
implementation of those activities, that has the most impact on students. 

METHOD AND SOURCES OF DATA: A 50-minute essay exam designed to assess 
writing skills was administered at the beginning of fall term and again at the end of 
spring term to students in composition classes at three colleges in the Minnesota 
Community College System. This resulted in pairs of pre-test and post-test essays from 
113 students, which were blind-scored by faculty trained in the use of holistic rating 
methods. The essay question had received extensive prior use in earlier WAC research; 
our sample was restricted to first-year students, however, so the question was one the 
students had not seen previously. Due to the difficulties in finding essay questions of 
comparable content and difficulty, the same question was used for both pre- and post- 



testing; we assume that the passage of nine months' time between pre- and post-testing 
rendered carryover effects negligible. 



To test hypothesis #1, a single-factor repeated-measures analysis of variance (ANOVA) 
was conducted to determine whether students improved the quality of their writing over 
the course of the academic year. Additional analyses were run to investigate how student 
academic background, ability (as reflected by GPA) and demographics might interact 
with this gain. 

To test hypothesis #2, a measure of instructional writing experience (a "WACiness" 
score) was calculated for each student as the number of non-composition courses that 
students took from WAC-trained instructors, plus the number of composition courses 
completed regardless of whether they were taught by WAC or non-WAC instructors. 
Interaction of students* WACiness score with their essay score (the repeated measure) 
was tested. 



A subsidiary hypothesis that the quality of instructional writing activities would also 
contribute to gains in the quality of students' writing, was explored in a pilot study. 
Faculty trained in holistic rating methods blind-scored instructional wiring assignments 
and descriptions of how these were implemented in the classrooms of instructors who 
collected student essays. These "Quality of Writing Assignment" scores were then 
compared to the means of students' essay ratings from those classes. Since the writing 
assignments used in the study were available for only 15 of the 37 instructors who 
collected essays, results were preliminary and suggestive. The pilot study will be 
described in some detail towards the end of this paper. 

Results will be interpreted within a context of findings from related cross-sectional and 
case studies conducted earlier in the grant. 

RESULTS: Hypothesis #1, that students would improve the quality of their writing over 
the course of an academic year, was confirmed. Post-test essay ratings averaged 
approximately one-half point higher than pre-test essay ratings on a 12-point scale. This 
result was statistically significant with p=.0177. This gain was quite consistent across 
GPA groups (null interaction p =.657). 

Hypothesis #2, that students' gains in quality of writing would increase according to the 
amount of their instructional writing experience, was supported at the 10% leveZ of 
probability. This result was enhanced by results from a cross-sectionr l study conducted 
earlier in relation to the same WAC program, which showed a significant (p <.01) 
positive correlation between amount of instructional writing and students' essay ratings. 

A curious aspect of this finding was that, whereas those who had the least number (2-3) 
and those who had the most number (6-12) of WAC or composition classes during the 
year gained about 0.75 point each, those having 4-5 classes showed no gains in their 



essay ratings. Could this be a plateau effeu? We found no academic (e.g. GPA) or 
demographic feature (e.g. age) that otherwise distinguished students taking 4-5 
WACfcomposition courses. 

There was no interaction between number of composition courses completed and rating 
gains. This implies that there is a WAC-specific effect - i.e. that the across-the- 
curriculum experience may be a crucial factor, and that we should not rely on 
composition courses alone to improve students* writing. 



COMPETING HYPOTHESES 

a) Concomitant instruction; Students who complete a large number of WAC courses 
undergo many instructional experiences apart from writing. It is possible that it is 
concomitant course instruction, aspects of the college environment, or maturation, rather 
than instructional writing experience, that contributes to improved writing. To see 
whether that could be the case, we examined the relationship between the absolute 
number of courses students completed (i.e. the number of non-WAC as well as WAC 
courses) and gains in essay ratings. Since statistical analysis yielded no interaction effect, 
we may conclude that concomitant instruction did not serve as a variable. 

b) Student characteristics: Since pre-test and post-test essays were completed by the same 
students, individual differences which might otherwise affect results were controlled. In 
addition, as reported above, students benefited from WAC experiences equally, 
regardless of GPA 

c) Instructor characteristics: Since participation in the WAC staff development program 
was voluntary, there was some concern that WAC faculty might be "better teachers" than 
non-WAC faculty, and that factors other than their use of instructional writing may have 
affected student outcomes. One thing that would dampen a potential volunteer effect is 
that the program involved approximately one-quarter of the 2,000 faculty in the 
Minnesota Community College System over the six-year grant period - a large and 
diverse group and an extended time-frame. Also, two case studies conducted earlier in 
the grant suggest that the amount, type, and quality of writing assignments instructors use 
are likely to be primarily responsible for the large variability of essay ratings among 
WAC classes. 

NEGATIVE APPLICATION CASE STUDY: A WAC-trained instructor used grammar 
study instead of instructional writing to teach a composition class. Essays were collected 
from this and 36 other classes as part of a cross-sectional study. A secondary analysis of 
essay ratings confirmed a negative hypothesis that the class would score significantly 
below the mean, despite the fact that the students had an above-average GPA There is 
a probability of less than 5% than the result could be due to chance. This case shows 
that it is not WAC training or the characteristics of participants per se that make a 



difference. Voluntary participation in the WAC program may attract better-than-average 
teachers; but if those teachers do not actually use instructional writing, their students* 
writing will not improve. 

POSITIVE APPUCATION CASE STUDY: An instructor new to the System specifically 
designed his history and political science classes according to the advice offered in WAC 
workshops and materials, and by the Keynote presenter Bill Coles. A political science 
class of his was used as a case study to see whether an exemplary application of WAC 
principles would result in higher essay ratings. The class received the highest mean 
rating of the 37 classes analyzed in the cross-sectional study, significant at the .01 level of 
probability. Although his class had an above-average GPA, his students still did 
significantly better on their essays even after a statistical adjustment was made for the 
difference in mean GPA among classes in the sample. This case shows the potential 
importance of course and assignment design and implementation. It provided the 
impetus for conducting a subsequent pilot study which would focus on instructional 
writing assignment variables. 



PILOT STUDY ON QUALITY OF WRITING ASSIGNMENTS 

The subsidiary hypothesis, that the quality of instructional writing assignments would have 
an effect on the quality of students* writing, was explored in a pilot study. A formal 
analysis was not conducted, because there was too much missing data and because there 
was insufficient time in which to perfect the procedure used for holisticalry rating the 
writing assignments. However, in our opinion, the hypothesis received sufficient support 
from the pilot and from other components of the WAC evaluation to recommend that 
further, intensive research should focus on the quality of writing assignment variables, 
and more broadly, on the quality of students' instructional writing experience within a 
course or progiam. 

Procedure: After scanning materials (syllabi, writing assignments, etc.) and/or 
information collected by phone from 27 WAC-trained instructors, 14 faculty trained in 
holistic rating methods constructed a scale which they would then use to rate the quality 
of instructional writing reflected by those materials. The scale is given in Table 1. 
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Table 1 



INSTRUCTIONAL USE OF WRITING IN THE CLASSROOM 



High 



6 - Very high quantity and probably high quality 
5 - High quantity and probably high quality 



Medium 



4 - Fair amount of writing with fair-to-good quality 
(materials provide examples which demonstrate quality) 
3 - Fair amount of writing but quality is questionable 



Low 



2 



1 



Little instructional writing, + quality is 

questionable 
No instructional writing was used 



Unratable X - Can't rate (not enough information) 



The raters then examined the materials more closely and independently gave each a 
score based on the scale descriptions. Materials were coded and instructor name, 
college, and other identifiers were eliminated to ensure a "blind" rating. 

Ratings of instructional materials were then tallied on the blackboard. Descriptions of 
instructional designs summarized from phone interviews were also rated, but except for 
instructors who were forthright in saying they used no or little instructional writing (#01, 
#03, #18, and #19 in Table 2), phone descriptions were generally deemed unreliable 
and are not included in this report. As you see from Table 2, there was a reasonable 
degree of preliminary consensus, especially considering that the materials were diverse in 
discipline and format, and that there was insufficient time for the group to polish the 
procedure and internalize the criteria. 



Table 2 



Materials 

Code 

#21 

#22 

#23 

#24 

#25 

#26 

#27/28 

#29 

#30 

#31 

#32 

#33 

#34 

#01 

#03 

#18 

#19 



High 
14 
10 
4 

3 
3 

7 
9 
10 
6 
8 



WRITING ASSIGNMENT RATINGS 
Medium 



(from phone interview) 



if 



4 

10 

6 

10 

7 

7 

5 

4 

8 

5 

7 



Low 



14 

3 
1 
7 



"little writing" 
'little writing" 
"no writing" 
"little writing" 



^signment ratings were then given an "eyeball comparison" with the mean essay ratings 
obtained by students from the classes in which the assignments were given. A more 
formal analysis did not seem warranted, given the draft quality of the pilot 



Table 3 



STUDENT ESSAY RATINGS 
(For top six and bottom six ranktd classes) 



Rank Order of Passes Represented by Materials 

Class Mean Instructional Materials Code Rating 

#1 #22 Hi 

#2 NA 

#3 NA * 

#4 NA 

#5 NA * 

#6 #21 Hi 



#32 #27/28 Med-to-Lo 

#33 #24 Lo 

#34 #33 Hi-to-Med 

#35 #29 Hi-to-Med 

#36 #18 Lo 

#37 #01 Lo 



Analysis and Findings: The 37 classes involved in an earlier cross-sectional study were 
rank ordered by mean essay rating. The top six and bottom six classes were examined to 
see whether they differed markedly in writing assignment ratings (Table 3). The classes 
ranked first and sixth were two of the three which received the highest assignment 
ratings. (Essay ratings were not available for the third.) We were unable to collect 
assignment materials from the other high-ranking classes. However, the fourth highest- 
ranking class was taught by an instructor who had had an assignment published in a 
WAC Handbook as an exemplar of good design. 

Assignment ratings were available for all of the six classes which received the lowest 
mean student essay ratings. The classes which ranked 33rd, 36th, and 37th on essay 
ratings were among the five given 'low" consensus ratings on assignment design. The 
class ranked 32nd used materials #27/28 in Table 2, which received medium-to-low 
ratings. The 34th-ranked class used materials #33, which as you see in Table 2, received 
high-to-medium ratings. However, that class (the 34th-ranked) was a developmental 
education class, with students who had the lowest mean GPA in the sample. 

The only class whose essay results were in clear contradiction to expectations was the 
35th-ranked class. That one's materials (#29 in Table 2) were rated high-to-medium; yet 
its mean essay ratings were third-lowest in the sample. The class ranked 26th in GPA - 
well below the mean for the sample, but not low enough to provide an excuse! 
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Materials #3, #19, #23, #25, #26, #30, #31, and #32 were unaccounted for by the top 
and bottom six classes. Essays were not collected from the classes which used materials 
#30 and #31, so we must exclude them from this analysis. Materials #23, #25, and #26 
received strong "medium" ratings; and materials #32 scored medium-to-high. Since these 
classes achieved mean essay ratings which ranked in the middle two-thirds of the 37 
classes in the sample, their materials scores were compatible with their essay scores. 

Materials #3 and #19 received "low" ratings. Although the chemistry and health classes 
which used these materials obtained essay ratings which ranked in the middle two-thirds 
of the classes involved, students* writing performance was lower than might be expected 
from their GPA's. The chemistry class had the highest mean GPA of those in the 
sample; yet the class ranked 25th on their essay ratings. The health class ranked 15th in 
GPA but 26th in essay ratings. 



CONCLUSIONS 

Results of the longitudinal study indicate that Writing Across the Curriculum programs 
have the potential to improve the quality of writing for students of all levels of academic 
ability. This conclusion is more convincing in the context of an earlier cross-sectional 
study conducted in conjunction with the same program. However, results from an 
accompanying pilot study and from earlier case studies suggest that the quality of writing 
assignments is likely to be more important than the number of writing assignments 
students are given. If this is true, WAC programs need to go beyond encouraging faculty 
to use instructional writing in their courses; they need to teach faculty how to design 
courses and writing assignments according to quality characteristics. To do that, we must 
have a better understanding of what these "quality characteristics" are. Further WAC 
research should concentrate on identifying the features of assignment design which 
optimize student learning, and on developing a systemmatic procedure for rating the 
quality of instructional assignments based on these features. Then and only then will we 
have a comprehensive basis for assessing and maximizing the effectiveness of WAC 
programs. 



8 



ESSAY QUESTION 



tm-class ESSAY: Read the quotation below and tha diractions for 
S* etsay SSt f OUO, ALL THE WAY THROUGH BEFORE you begin your 

». ! You may mak. a scratch ^utUna ^writa f ro«, ^ you 
need to paca yoursalf so that you COMPLETE THE ESSAY DURING THIS 
CLASS SESSION: Usa your bast English so that you do NOT naad to 
recopy your assay before you hand it in. Cross-outs and minor 
editing ara accaptabla. 

"We all lova to instruct, though ws can 
teach only what is not worth knowing. 

This statement raises several questions about the nature of 
learning and the value of what can be learned and taught. 

Think about YOUR attempts to "teach" someone something. Your 

actually haS. bi." * t.ach«r of • claaa, a camp counjalor. o* 
tutor! Perhaps you taught th. child you bai»yiat to shoot 

baskets . 

CHOOSE ONE SPECIFIC INSTANCE in which you ware a "teacher" and 
write an essay in which you: 

DESCRIBE the experience clearly. 

---EXPLAIN to what extent you were able to convey what you 
understood or what you knew how to do. teach? WHY 

(what COULD you teach? What COULDN * knowing"? 

max ara^fiuwyssr u-sm 

teaching and learning. 



I Revised Feb. f 87 ) 

WAC EVALUATION PROJECT 

HOLISTIC SCORING GUIDE FOR AUSTEN QUESTION 

in scoring this assignment f w« should reward studsnts 
for what they have dons well, keeping in mind that this is 
first -draft writing on an assignment which they have just 
received* 

Students ere asked to do three things: i ) to describe a 
situation in which they taught something; 21 to explain the 
extent to which they were able to convey what they were teaching 
(Note: good writers will also evaluate!} and 3) to discuss how 
the given quotation might apply to what one can learn* These 
tasks do not have to be handled separately , but may be integrated 
in the organisation of the eeeay. 



UPPER-HALF PAPERS: 

The *$« paper has all thm characteristics of the "5 
paper, except that it will make more effective use of the 
quotation. It will also ba etyllsticaliy impressive, exhibiting 
unity of tone and point of view, graater depth of analysis, and 
some originality* There will be no distracting mechanical errors 
in this pa pgr* 

THE PAPER completes all the tasks, although thm 
discussion of the quotation may be leas well -developed than thm 
description and explanation. Using specific details, the assay 
will describe a particular kind of teaching expanence, not 
necessarily a single instancm of teaching. The axp la nation will 
go beyond a narrow, literal reading of the assignment and mil 
lead into an evaluative discussion of the quotation. The 
discussion will show an understanding of thm quotation and its 
relationship to what one can learn? the student »ay deny the 
applicability of the quotation. Thm paper will have clear 
organisation, good transitions, and will be generally free from 
distracting errors of grammar, diction, spoiling and punctuation. 



The *4" papar haa a good description and at least an 
adequate evaluation of the taachinq experience, but thm 
discussion of thm quotation may be wma*. This papar will be lass 
coherent than thm m S" paper, with unclear or missing transitions. 
The mechanics will ba competent but smudged by a few noticeable 
errors* 



LOWER -HALF PAPERS 5 ^ 

Even though it may address all of the assigned tasks, the 
"3" paper fails to complete one or more of them adequately; OR 
it will ba disorganised, lacking direction and tranaitions; or 
the mechanics will be muddy (tortured syntax, careless diction, 
poor spelling and punctuation). 

THE "2" PAPER attempts to fulfill the assignment but does 
not actually completa any of thm three taekaj OR it will ba 
disorganised; OR thm mechanics will be filthy* In term; of not 
adequataly addressing the tasks, it is likely to 
narrow, literal reading of the aaeignmmnt. It may DESCRIBE an 
experience at graat length without EVALUATING ita implicetiona 
and ita relationship to thm quotation* 

The w l w papar attempts to fulfill the assignment but 
does not actually complmte any of the threa tasks, AND it will bm 
disorganised; AND it will have filthy mechanics. 




"X" PAPERS! 

Th* "X" psp*r m*K«* no *tt*»pt to do th* •••i<jn**nt ; th* 
stud*nt hat *rttt*n on *noth*r topic. Such • p*p*r should **t 
thi* »cor* no »*tt*r ho. **U-writt*n it a*y b*. 
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